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Abstract 

Background: Developing a clinically relevant set of quality measures that can be effectively used by an electronic 
health record (EHR) is difficult. Whether it is achieving internal consensus on relevant priority quality measures, 
communicating to EHR vendors' whose programmers generally lack clinical contextual knowledge, or encouraging 
implementation of EHR that meaningfully impacts health outcomes, the path is challenging. However, greater 
transparency of population health, better accountability, and ultimately improved health outcomes is the goal and 
EHRs afford us a realistic chance of reaching it in a scalable way. 

Method: In this article, we summarize our experience as a public health government agency with developing 
measures for a public health oriented EHR in New York City in partnership with a commercial EHR vendor. 

Results: From our experience, there are six key lessons that we share in this article that we believe will 
dramatically increase the chance of success. First, define the scope and build consensus. Second, get support from 
executive leadership. Third, find an enthusiastic and competent software partner. Fourth, implement a transparent 
operational strategy. Fifth, create and test the EHR system with real life scenarios. Last, seek help when you need it. 

Conclusions: Despite the challenges, we encourage public health agencies looking to build a similarly focused 
public health EHR to create one both for improved individual patient as well as the larger population health. 
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Background 

In 2004, in an effort to help New Yorkers live longer 
and healthier lives, the New York City Department of 
Health and Mental Hygiene (DOHMH) undertook a 
policy initiative called Take Care New York (TCNY). 
The goal of TCNY is to improve population health by 
helping New Yorkers take discrete and identifiable steps 
to improve their health through focusing on ten items 
ranging from prevention to health screening and chronic 
disease management. After identifying these ten areas 
(see Table 1), the DOHMH, guided by then Commis- 
sioner Thomas Frieden, tracked these TCNY measures 
to monitor the city's progress [1]. In early 2005, the 
Commissioner also set up a Taskforce at the DOHMH, 
the Primary Care Information Project (PCIP), tasked 
with reducing health care disparities among New 
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Yorkers, especially those living in lower socioeconomic 
neighborhoods, through health information technology. 
PCIP was established to help low-income, disadvantaged 
communities and the providers who deliver health care 
to these communities take advantage of technological 
innovations in order to improve the quality of care of 
their patients as measured by the ten TCNY measures 
[2]. Recognizing the opportunity to improve care and 
collect data on the TCNY defined clinical areas, PCIP, 
in collaboration with our selected EHR vendor, devel- 
oped the corresponding TCNY quality measures for use 
at the point of care. These measures at the point of care 
became the basis of our clinical decision support sys- 
tems (CDSS). Within the EHR, these CDSS alerts inform 
the provider by a series of prompts on their computer 
screen that the patient they are currently seeing in their 
office is due for specific health maintenance screenings 
or chronic disease management based on the patient's 
age, gender, and comorbidities. 
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Indicator 

Adult New Yorkers without a regular doctor 

Adult New Yorl<ers who smoke 

Proportion of New Yorkers with high blood pressure and cholesterol. 
Proportion of New Yorkers with well-controlled hypertension, cholesterol & diabetes 

Number of New Yorkers who die from HIV/AIDS 

Prevalence of untreated depression 

Alcohol-attributable mortality 
Drug-related deaths 

Screening rates for breast cancer 
Screening rates for cervical cancer 
Screening rates for colon cancer 

hfluenza immunizations among New Yorkers age 65+ 

Women who die from intimate partner homicide 
Children with newly-identified blood lead levels (BLL) > 15 pg/dl and an identified lead-based 

paint hazard 

nfant mortality rate per 1, 000 live births 



Table 1 Summary Table of Take Care New York Indicators 

TCNY Agenda Item 

1. Have a Regular Doctor or Other Health Care 
Provider 

2. Be Tobacco Free 

3. Have a Healthy Heart 

4. Know Your HIV Status 
5. Get Help for Depression 

6. Live Free of Alcohol or Drugs 

7. Get Checked for Cancer 

8. Get the Immunizations You Need 

9. Make Your Home Safe & Healthy 

10. Have a Healthy Baby 



This collaborative development between a govern- 
ment entity and a private EHR vendor resulted in a 
unique public health oriented electronic health record, 
which has actionable quality measures that prompt 
providers to take steps for preventive care or chronic 
disease management for various health issues targeted 
within New York City. A public health oriented EHR is 
an electronic health record system which in addition to 
enabling the provider to provide appropriate healthcare 
resources to the care of their patients, allows for safe, 
convenient, and timely transfer of public health infor- 
mation bilaterally between providers and their local 
and state public health organizations and departments. 
To contribute to the measurement of the TCNY mea- 
sures, the aggregate values of the corresponding mea- 
sures are transmitted monthly to DOHMH. Analysis of 
such data allows for improved efficacy of important 
public health programs such as infectious disease sur- 
veillance, chronic disease prevalence studies, and envir- 
onmental exposures, to name a few. This method of 
gathering data from community providers has the 
potential of eventually replacing the traditional slow 
and incomplete methods of data collection, such as 
chart reviews, which public health agencies have been 
using for more than 100 years. Since we developed this 
public health oriented EHR, we have implemented the 
system to approximately 2600 providers in the five 
boroughs of New York City representing approximately 
2.5 million patients. In 2010 alone, our CDSS alerts 
have been involved in million patient encounters. 
While the EHR development process has been long, 
the details of the journey may help illuminate the 
pathway for other local or state governments with a 
similar interest. 



Methods 

PCIP considers electronic health records to be one of 
the best and most scalable methods of helping commu- 
nity providers improve the care they provide while con- 
taining providers' operational costs in the long term 
after an initial upfront investment [3,4]. EHRs allow 
documentation of a clinical encounter in a structured 
codified format with electronic ordering and recording 
of laboratory tests, radiologic examinations, immuniza- 
tions, vital signs, medication and allergies to name but a 
few important functionalities necessary for a clinical 
encounter. Structured data is the crux of public health 
functionality because it enables: quality measures to 
assess ambulatory care quality, CDSS to communicate 
best clinical practices to providers, and public health 
interfaces to link with existing DOHMH systems. All 
three of these functionalities are at the core of an ideal 
public health-oriented EHR, and all three are enabled by 
structured data. At the time of PCIP's establishment, 
there was no EHR that included the public health func- 
tionality that we envisioned. 

Before implementing EHR systems in the community, 
the DOHMH tried to take advantage of existing techno- 
logical innovations to improve the disparity of health 
care through the distribution of personal digital assis- 
tants (PDA) to physicians working in federally-funded 
community health centers [5]. The PDAs were primarily 
used for electronic prescribing, while health alerts and 
information on emergency preparedness were available 
for download via a website accessible from the devices 
and maintained by an outside vendor. These health 
alerts were an early, primitive form of CDSS. They 
informed providers of potentially relevant information in 
hopes of helping them make an informed clinical 
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decision. However, even though vendor-led training was 
conducted during the implementation phase, it quickly 
became apparent that adoption even of a seemingly 
easier technological implementation than a full electro- 
nic health record can be difficult, particularly when the 
methodology used is outside of the standard workflow 
of a physician [6-8]. When the complexity of imple- 
menting PDA-based CDSS to providers approached the 
anticipated difficulty of implementing EHR-based CDSS, 
we decided to concentrate our efforts on a more com- 
prehensive EHR in order to gain the added benefits over 
a simple electronic prescribing software. This initial pro- 
ject provided us the insights to many of the barriers and 
complexities of technological implementations in clinical 
settings such as provider comfort with new and existing 
technology, software and hardware support and mainte- 
nance required, length and importance of user training, 
and provider and staff "buy in" in the process. As a 
result, we sought to identify an EHR vendor that would 
collaboratively develop our set of public health 
requirements. 

In January of 2006, the DOHMH released a Request 
for Proposals (RFP) to identify a qualified vendor to pro- 
vide an electronic health record system to our New 
York City community-based providers as well as to Cor- 
rectional Health Services, the City entity responsible for 
providing health care in the New York City's jail system. 
After the release of the RFP, six vendors were invited to 
provide a live demonstration of their product to the 
DOHMH product review committee, made up of the 
Assistant Commissioner of PCIP, Chief of Operations of 
PCIP, Director of Finance and Operations of Health 
Care Access and Improvement, and Director of Vendors 
and Technology Partners. The criteria used in the ven- 
dor selection process included the size of the company, 
years the company has been in business, stability of the 
company determined by a stable leadership team over 
time, the usability of the product as determined through 
impromptu case presentations to the vendor, ease of 
use, adaptability, and the interest and resources available 
to the company to make the necessary changes that we 
were seeking. The selection of eClinicalWorks as the 
awardee of the four-year contract was announced in the 
fall of 2006. eClinicalWorks was chosen because of their 
commitment to being our partner in building an afford- 
able public health-oriented EHR. After selection, we 
soon began development discussions in which we com- 
municated necessary public health requirements to the 
eClinicalWorks developers. 

Once the EHR vendor had been identified, PCIP 
defined the necessary public health requirements and 
categorized them into two development cycles. The first 
development cycle would produce the CDSS alerts as 
well as the corresponding order sets which would 



facilitate providers ordering. In addition, during the first 
development cycle, a more robust registry system that 
would allow providers to assess their patient population 
on demand would be developed. The first development 
cycle involved three phases, which are pre-development 
discussions, collaborative development, and quality 
assurance [see Table 2]. The second cycle focused on 
developing functionality that will allow the DOHMH to 
distribute public health alerts during important events, 
such as infectious disease outbreaks. 

In parallel to working on the Request for Proposals, 
PCIP started work on translating the ten TCNY priority 
measures into actionable CDSS alerts at the point of 
care. These CDSS alerts have been defined as "an auto- 
mated process for comparing patient-specific character- 
istics against a computerized knowledge base with 
resulting recommendations or reminders presented to 
the provider at the time of clinical decision making" [9] 
That is, making known to the physician a patient's qual- 
ity measure status at the point of care is the decision 
support. We expanded this standard function of CDSS 
by making it actionable for our providers so that not 
only are providers aware of the applicable alert, they can 
also address the clinical need with one click of a compu- 
ter mouse. However, the first step in producing these 
new alerts was to investigate the existing measures at 
that time and see how they were consistent with the 
standards produced by other national and state 
organizations. 

Our approach to developing electronic CDSS mea- 
sures involved consulting existing national bodies as 
well as internal DOHMH subject matter experts in 
order to translate their retrospective paper measures to 
point-of-care electronic measures. We examined exist- 
ing measures produced by national organizations such 
as the Physician Quality Reporting Initiative and 
National Committee for Quality Assurance as well as 
measures produced by independent national and pro- 
fessional medical associations such as the U.S. Preven- 
tive Services Task Force, American College of 
Cardiology, American Diabetes Association, and Amer- 
ican Lung Association. 



Table 2 Phases of Development Cycle 



Step 


Goal 


Pre-development 


■ Involve stakeholders 




■ Build consensus 


Collaborative 


• Plan frequent meetings 


Development 






■ Develop clinical to programmatic 




translation 


Quality Assurance 


• Ensure alpha testing 




• Provide beta and clinical content testing 
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In addition to reconciling with national standards, we 
invited DOHMH subject matter experts on sexually 
transmitted diseases, tobacco, asthma, and mental health 
to get their input on the respective measures and to 
devise the best method of data collection. We met with 
health organizations within New York City to get their 
recommendations on a practical set of measure alerts. 
For example, PCIP worked with subject matter experts, 
informatics specialists, and clinical providers from the 
Institute for Family Health, a community health center 
based in New York City, to develop our measures. 
These measures needed to incorporate clinical guide- 
lines, population level epidemiologic information, and 
optimal design based on extensive clinical experience to 
ensure that they were not only clinically appropriate and 
useful to clinicians but were also consonant with exist- 
ing workflows to ensure their use. At the end of this 
one year process and after incorporating the recommen- 
dations of all these subject matter experts, we had trans- 
formed the 10 TCNY general health areas into 40 
specific quality measures at the point-of-care CDSS 
alerts (see Table 3). 

Once these 40 clinical quality measures were agreed 
upon, the PCIP development team undertook the task 
of translating these 40 measures into logic models that 
could be processed and interpreted by a computer pro- 
gram. The logic models were specific actions and pro- 
cesses we required the computer program to run for 
every patient depending on a variety of demographic 
and clinical content entered by the provider in the EHR. 
For example, if the patient was over 18 years old, we 
required the program to identify if smoking status was 
documented for the patient and if it had, to determine if 
the last status of "never", "former", or "current smoker" 
had been updated within the past year. The creation of 
the logic models was a good method that allowed the 
team to identify how the EHR system could determine 
the particular measures a patient qualified for. It also 
helped formulate in our minds the necessary data ele- 
ments that were crucial for a particular measure. In 
addition, the process of creating the logic models was 
successful in building consensus across subject experts 
on how a measure can be interpreted proactively at the 
point of care and how to reconcile best practices with 
small practice workflow. 

However, translating guidelines at the point of care 
often resulted in these logic models becoming extremely 
complicated. For example, the EHR software would have 
to keep track of each patient through the logic diagram, 
save their progress, and retrieve it quickly at the next 
clinical encounter. Moreover, the system was then 
required to keep track of several conditions and to use 
data for multiple "if-then" statements. It quickly became 
apparent that this longitudinal examination of the 



patient was too complicated and these difficulties with 
complex coding would result in slower system perfor- 
mance. A limitation of the logic models was that its use 
did not allow for comparative analysis across patients 
and providers. With logic models, it was simply a matter 
of where a patient was placed in the flow diagram; there 
was no inherent "good" or "bad" state for the patient to 
be evaluated on. While logic models are very good for 
providing decision support for one patient at the point 
of care, there was no easy way to determine a group of 
patients' compliance to a particular measure. In other 
words, while the alerts were appropriate and timely, it 
was difficult to evaluate the compliance rate to these 
alerts by patient and provider. Despite the complexity 
and ultimate abandoning of the logic model concept, 
the method helped to elucidate the data elements neces- 
sary to calculate quality measures solely based on EHR 
derived data. 

In response to the challenges presented by the logic 
model, the team shifted our philosophy to aggregate 
counting and developed compliant binary counts. Com- 
pliant binary counts are defined as true/false assess- 
ments of whether a patient meets or does not meet the 
defined criteria, thus indicating that a patient is either 
compliant or non-compliant for a particular measure. If 
a patient meets the criteria for disease evaluation, then a 
"I" is placed in the denominator. If a patient meets the 
criteria for measure satisfaction, then a "1" is placed in 
the numerator. Therefore, every patient for each applic- 
able measure is either, "0/1" or 0% for non-compliant or 
"1/1" or 100% for compliant. For example, one of our 
measures asks that every patient with diabetes mellitus 
have their hemoglobin Ale checked at least once every 
6 months and the result to be entered in the EHR. If a 
patient is documented in the EHR as having diabetes 
mellitus as a medical condition in their problem list, 
then they are added to the hemoglobin Ale testing mea- 
sure and the denominator of that measure increases by 
1. If that provider then orders a hemoglobin Ale test 
for the patient and the result of the test enters into the 
EHR, the numerator increases by 1 and the patient is 
fully compliant (1/1) for this measure. If not, the patient 
is not compliant (0/1). If the provider has 500 diabetic 
patients, this number can be summed for a compliance 
rate for this measure. If 400 hemoglobin Ale lab tests 
results are eventually entered for these patients, then 
the provider's compliance rate will be 80%, indicating 
that 80% (400/500) of the provider's diabetic patients 
have had the appropriate intervention for this measure. 
In this way, results from a group of patients can be 
aggregated by provider, thus providing a quality assess- 
ment of the provider's patient panel which for this parti- 
cular provider for this particular CDSS alert would be a 
compliance rate of 80%. The major advantage of using 
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Table 3 All Alerts are on an annual basis unless specified otherwise 



TCNY Domain 



Measure Name 



Brief Description of Measure 



Have a regular doctor 

Be tobacco free 



Keep your heart healthy 



Know your HIV status 
Get help for depression 



Live free of dependence on alcohol 
and drugs 

Get checked for cancer 



Get the immunizations you need 



Make your home safe and healthy 



Have a healthy baby 



Patients see assigned PCG 

Smoking status 

Smoking cessation 
intervention 

BP control in HTN 

Antithrombic 

Body IVlass Index 

Cholesterol screen 

Cholesterol control 

LDL control (high risk) 

LDL testing (high risk) 

AlC testing 

A1C control (< 7%) 

BP control in IVD 

BP control in IVD AND 
HTN 

BP control in DIV 

BP control in DIV AND 
HTN 

HIV screening 
HIV viral load and CD4 
Depression screening 
Depression follow-up 
Depression control 
Alcohol use screening 

Colorectal cancer 
screening 

Breast cancer screening 

Cervical cancer screening 

Influenza vaccine (child) 

Influenza vaccine (high 
risk) 

Influenza vaccine 

Pneumococcal vaccine 

Lead testing (1 year) 

Lead testing (2 years) 

Asthma symptom 
assessment 

Asthma control 

Chlamydia screening 
Sexual history 
Sexual history taken 



See assigned PCG at least once 
Update smoking status 

Stop smokers with medications or counseling 

Control BP < 140/90 in patients with HTN without DIVl or IVD 

Provide Patients with IVD or DM with antithrombic therapy 

Measure the height and weight for BMI 

Screen patients without DM or IVD for cholesterol/HDL 

Control patients without DM or IVD to a total cholesterol < 240 or LDL < 160 

Control patients without DM or IVD to an LDL < 100 

LDL screen for patients with DM or IVD 

Measure at least one HbAlc in DM patients every 6 months 

Control HbAlc below 7.0% in DM patients 

Control BP < 140/90 with a diagnosis of IVD without DM 

Control BP < 140/90 with a diagnosis of IVD AND HTN without DM 

Control BP < 130/80 with a diagnosis of DM 

Control BP < 130/80 with a diagnosis of DM AND HTN 

Screen for HIV 

Monitor HIV patients' viral load and CD4 every 3 months 

Screen patients for depression 

Monitor patients with depression every 3 months 

Lower depressed patients PHQ-9 score 

Screen for alcohol misuse 

Screen 50-80 years old patients 

Screen female > 40 years old 
Screen female 18-64 years old 
Vaccinate children 7 months to 5 years old 
Vaccinate high risk patients 5-49 years old 

Vaccinate > 50 years old 

Vaccinate > 65 years old or ( > 60 months AND in a high risk group) 
Screen one-year old with a blood lead test 
Screen two-year old with a blood lead test 
Assess 18-56 years old for asthma severity 

Assess 5-56 years old with persistent asthma who were prescribed appropriate 
medications 

Screen eligible female patients 
Update > 18 years old sexual history 
Update 12-17 years old sexual history 



binary counts is that it allows a quantitative assessment 
of each patients' compliance to a particular CDSS alert 
and allows for comparison of providers' performance to 
city or zip code averages. 

In order to provide some lag time between the time 
the provider orders a lab and the result return from the 
commercial laboratory, we also introduced the concept 
of snoozing alerts. It is a transition phase allowing the 



patients a certain amount of time (usually one month) 
to get their labs drawn and for the result to return to 
the provider's EHR. A snoozed alert disappears from a 
provider's CDSS menu but will return as an alert if the 
lab result does not appear in the time window. 

Once measures had been defined and we had agreed 
on how the DOHMH and our providers would use 
them to monitor and improve important quality 



Amirfar et al. BMC Public Health 201 1, 11:753 
http://www.biomedcentral.com/1471-2458/11/753 



Page 6 of 8 



indicators, PCIP began the development phase in which 
we created an overall plan for customizing the electronic 
health record to meet the needs of New York City pro- 
viders. We developed a timeline for the project and 
defined our measures for success, namely 2000 providers 
being live with a full public health oriented EHR with 
actionable CDSS in two years. Our long term goal has 
always been to improve clinical outcomes for New Yor- 
kers and all our project plans were produced with this 
overarching goal in mind. 

The next crucial step in this phase was producing our 
CDSS alerts within the software by translating the mea- 
sures with the binary counts we had produced with our 
DOHMH and community partners into definitions that 
computer software could understand. To do this, we 
had to create a PCIP team dedicated to producing this 
new system. Our development team consists of quality 
assurance specialists who test the new product, EHR 
super users who go to the providers and provide tailored 
training, physicians who provide clinical input and prac- 
tice workflow experience, epidemiologists who analyze 
the resulting data, and project managers who coordinate 
all the activities. Once the team had been established, 
the operational structure for collaborative development 
between PCIP and eCW was created. We agreed that 
meetings between ourselves and eCW software develo- 
pers had to be frequent and lengthy in order to solidify 
relationships and to make sure we had similar expecta- 
tions of the final product. During these meetings, we 
spent the majority of the time defining the scope for 
each new proposed functionality as well as the rationale 
behind its introduction. We felt it was important that 
the vendor had "buy-in" for each functionality and felt 
that the proposed changes were necessary as well. Hav- 
ing senior executive leaders with both vision and con- 
tent matter expertise closely involved was critical 
because it created an environment where both the 
DOHMH and vendor could overcome the inevitable 
obstacles. The cooperation and shared vision helped 
both organizations to succeed and achieve their respec- 
tive goals. 

Even with the time spent describing each new func- 
tionality during the collaborative development meetings, 
there were many follow-up discussions for clarifying or 
redeveloping functionalities because items that eCW 
developed were not always what we had in mind. We 
quickly came to realize another important aspect of our 
relationship. While we might be agreeing on the facts, 
the interpretation can also make or break a function in 
terms of usefulness, effectiveness, and adoption. The 
software developer approaches each solution with a 
practical engineering viewpoint that must be countered 
with a clinical or public health viewpoint for balance. 
Because of this, it is very important to get frequent 



visual updates for each functionality via demonstrations 
to make sure the vision does not veer off the path from 
what is expected. 

One of the issues that our collaborative development 
team needed to discuss was the use of technological 
standards. Standards are necessary in an environment 
where the software needs to exchange medical content 
entered in the progress notes across many users. The 
software vendor needs standards so that their system is 
able to communicate between different users. For the 
DOHMH, our scope is broader since we need standards 
to satisfy our larger health information technology needs 
in order to receive aggregate data transmissions of clini- 
cal information both among all electronic health users 
in New York City and to the DOHMH. Since the deci- 
sion support tools were implemented for all our provi- 
ders, it was necessary to discuss the standard set of data 
the software would utilize to produce the alerts with the 
EHR vendor. Some standards already existed and we 
simply used them in our CDSS programming. For exam- 
ple, we used the International Classification of Diseases 
{ICD-9) codes as the standard for codifying the patient's 
medical problems and Logical Observation Identifiers 
Names and Codes (LOINC) codes for lab results return- 
ing from commercial laboratories. For other data ele- 
ments where no standard exists, it was necessary to 
make our own. Laboratory ordering was one area where 
there are no established standards and we made our 
own numerical classification system called community 
IDs. For example, an HIV test ordered from the system 
is the same irrespective of the commercial laboratory 
where the test is ordered. 

Results 

Once the different CDSS functionalities had been devel- 
oped in the electronic health record, testing the func- 
tionality began in earnest. We had underestimated the 
time and effort that testing of the functionalities would 
require. In fact, testing of the measures required more 
resources and was more complex than the development 
phase. Between physically entering test patients, ensur- 
ing that CDSS measures were functional, and keeping 
track of the bugs and versions developed, we quickly 
realized we had underestimated this portion of CDSS 
development. 

Testing of the various bugs within the system compro- 
mised two main stages. The first stage involved making 
sure that the functionalities of the CDSS system were 
working. This included alpha testing which entailed 
making sure new CDSS functions in the system worked 
such as quick links to labs and medications, and that 
there were no gross and obvious system errors. One of 
the most common errors we discovered during this part 
of the testing were usually HTML or JavaScript errors 
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where the CDSS alerts were supposed to be. As these 
areas in the system were slowly corrected, we focused 
our attention on the second stage of quality assurance. 
In the second stage, we primarily focused upon correct- 
ing any clinical content errors such as clinical alerts that 
appeared for the wrong patients using use cases. For 
example, if a test hypertension patient in the system had 
a blood pressure of 135/85, a blood pressure alert would 
not appear for this particular patient. However if a diag- 
nosis of diabetes was entered for this patient, their 
blood pressure would qualify them for the correspond- 
ing more stringent blood pressure alert for a diabetic 
patient. 

Conclusions 

Building meaningful measures as CDSS and incorporat- 
ing them effectively into an electronic health record was 
more challenging than anticipated. The steps involved 
are difficult and long and really cannot be shortchanged. 
All view points, including those of public health, clinical, 
and technical must be represented so that there is con- 
sensus. Since these measures will affect and direct provi- 
der workflow, the chosen measures must be clinically 
significant. There needs to be some understanding of 
where these measures fit into the other state and 
national measures that have already been developed. 
Once the measures are finalized, the difficult part is to 
translate them into a format that a computer program 
can understand and utilize effectively. Public health and 
quality measurement experts must find their counter- 
parts in software engineers who share a public health 
vision and will prioritize the development and imple- 
mentation of these measures. During their development, 
there need to be countless conversations to facilitate 
transparent translation of the measures from the clinical 
perspective to an engineering/programming perspective. 
Finally, testing of the measures needs to be done to 
make sure that they work at the right time for the right 
patient to give the right message. 

Having completed this process, there are some recom- 
mendations we can make to improve the chances of suc- 
cess for others seeking to build CDSS within their EHR. 
First, organize a workgroup of various clinical experts to 
help define the scope and build consensus. Sometimes, 
you will not know how big a project you are facing until 
you can see the challenge from every perspective. A large 
pool of experts will help with this. Not only will a large 
pool of experts allow for a wider viewpoint on clinical 
measures complying with both domestic and interna- 
tional standards but will also help the project to become 
a success since more people will have a vested interest in 
the project. Second, get support for your activities from 
executive leadership. The full support of two sequential 
City Health Commissioners and the Chief Executive 



Officer of our partner software company has been invalu- 
able for getting help and cooperation from not only our 
respective organizations but from the community as well. 
Third, find an enthusiastic and energetic software partner 
who shares your vision for your project and is well pre- 
pared to undertake a complex task such as this. The 
energy in a good relationship can be used to pull both of 
you through during the inevitable tough times. Fourth, 
implement a transparent operational strategy so both 
sides can understand what the other is developing. The 
same scope and function may be agreed upon in meet- 
ings but its actual implementation will inevitably vary. 
An engineer will always find ways of interpreting facts 
differently from what a clinician envisions. Do not 
assume anything except that they will see things differ- 
ently. It would be unfortunate to find out several months 
into development that the vendor is producing a func- 
tionality that one cannot use. Fifth, create and share use 
cases. You need to make sure that the right alerts show 
up for the right patients at the appropriate time. Testing 
by users and developers guarantees that the new func- 
tionality is working properly, but only a clinician can ulti- 
mately decide if the software "makes sense" in a clinical 
setting. Lastly, respect your time and effort in doing this 
project and ask for help if you need it. Others have gone 
through the process, so there is no need to reinvent the 
wheel. Guidance and advice from others can go a long 
way to getting things back on track. By following these 
recommendations, one can improve their chances for 
developing a clinically useful CDSS alerting system while 
minimizing the inevitable obstacles and frustrations. We 
have followed these recommendations ourselves in a later 
phase while developing new functionalities. The process 
has been much improved: shorter, more focused, and 
more satisfying for both parties. These efforts of develop- 
ing a public health oriented CDSS has allowed us to com- 
municate more effectively with providers when they are 
seeing patients and has advanced the relationship 
between local government agency and the providers of 
healthcare into the 21''' century. 
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